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ABSTRACT 

This  report  demonstrates  the  application  of  a  Bayesian  Belief  Network  de¬ 
cision  support  method  for  Force  Level  Systems  Engineering  to  a  collection  of 
projects  related  to  Aerospace  Surveillance  and  Battle  Management.  The  pro¬ 
posed  method  is  used  to  determine  the  most  credible  combinations  of  these 
projects,  and  the  most  plausible  evolutions  of  the  force  structures  that  they 
deliver  over  time.  Since  the  analysis  is  based  on  strictly  limited  information, 
the  results  that  are  obtained  are  of  a  highly  tentative  nature.  Nevertheless, 
the  potential  utility  of  such  an  approach  is  convincingly  demonstrated. 
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A  Belief  Network  Decision  Support  Method  Applied  to 
Aerospace  Surveillance  and  Battle  Management  Projects 


EXECUTIVE  SUMMARY 

The  problem  of  developing  and  sustaining  a  highly  capable  defence  force  to  meet  the 
diverse  needs  of  a  medium-sized  democratic  nation  such  as  Australia  is  an  extremely  com¬ 
plex  onerous  one,  particularly  in  view  of  the  plethora  of  challenges  and  ambiguities  posed 
by  the  political  circumstances  of  the  twenty-first  century,  the  rapid  rate  of  technological 
advancement  of  military  systems,  and  the  relatively  modest  level  of  funding  available.  One 
way  of  maximising  the  potential  performance  of  a  defence  force  is  to  ensure  that  its  as¬ 
sets  are  capable  of  cooperating  cohesively  together  as  a  highly  effective  system-of-systems. 
Achievement  of  this  goal  may  be  fostered  through  the  application  of  suitable  Force  Level 
Systems  Engineering  (FLSE)  principles. 

A  particularly  salient  feature  of  systems-of-systems  is  the  presence  of  multiple  stake¬ 
holders  and  decision-makers.  Sophisticated  and  robust  force-level  systems  design  methods 
that  can  effectively  match  the  needs  of  disparate  stakeholders  to  the  potentially  available 
options  are  required  in  order  to  alleviate  the  task  confronting  decision-makers.  However, 
certain  results  from  social  choice  theory  and  game  theory  have  implications  for  the  ways  in 
which  stakeholders’  preferences  may  validly  be  aggregated.  In  addition,  the  networked  na¬ 
ture  of  systems-of-systems  renders  evaluation  methods  that  rely  on  a  strictly  hierarchical 
decomposition  of  the  decision  problem  unsuitable. 

These  two  aspects  suggest  that  belief  network  methods  may  be  more  suitable  for  treat¬ 
ing  systems-of-systems  questions  than  are  more  familiar  evaluation  methods.  The  aim  of 
this  report  is  to  illustrate  how  certain  Bayesian  Belief  Network  techniques  might  be  ap¬ 
plied  to  FLSE  design  decisions.  The  method  is  intended  to  be  compatible  with  a  wide 
variety  of  project  management  and  systems  engineering  process  models,  and,  thus,  there 
is  no  particular  preferred  process.  However,  some  specific  features  that  a  process  suitable 
for  treating  systems-of-systems  questions  should  possess  are  mentioned  in  passing. 

In  general,  the  procedure  involves  eliciting  the  needs  of  the  stakeholders  and  then 
suggesting  possible  solutions  that  seem  to  best  conform  to  those  needs  for  closer  consid¬ 
eration  by  the  stakeholders  and  decision-makers,  by  means  of  automated  reasoning.  The 
particular  case  that  is  considered  as  an  example  here  is  that  of  Aerospace  Surveillance 
and  Battle  Management  (ASBM) .  A  number  of  legacy  assets  that  form  the  current  ASBM 
system-of-systems,  and  projects  which  will  transform  the  current  asset  configuration  into 
its  future  form,  have  been  identified  and  form  the  basis  of  the  analysis.  The  proposed 
method  is  used  to  determine  the  most  credible  combinations  of  the  projects,  and  the  most 
plausible  evolutions  of  the  force  structures  that  they  might  deliver  into  service,  over  time. 

As  it  may  reasonably  be  assumed  that  it  is  only  ever  necessary  to  actually  commit  to 
an  option  for  the  first,  impending  decision  epoch,  selection  of  the  most  credible  option 
for  the  first  decision  epoch  over  the  entire  ensemble  of  asset  configuration  trajectories  is 
more  appropriate  than  assuming  that  the  most  credible  trajectory  will  always  be  the  one 
to  ensue.  A  decision  that  is  made  in  this  way  will  be  termed  a  best  prospects  decision, 
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since  it  attempts  to  cater  for  a  wide  range  of  possible  future  developments.  This  may  in 
general  be  different  from  the  initial  choice  on  the  most  credible  trajectory. 

The  rationale  for  preferring  the  best  prospects  option,  rather  than  the  initial  choice  for 
the  most  credible  trajectory,  is  that  this  should  usually  result  in  there  being  more  relatively 
high  credibility  alternative  trajectories  on  into  the  future,  thus  providing  greater  flexibility 
as  stakeholder  needs  and  preferences  undergo  future  revision.  Consequently,  the  best 
prospects  decision  sacrifices  some  initial  credibility  about  the  alternative  that  is  selected 
in  the  interests  of  greater  future  flexibility.  Once  the  best  prospects  decision  has  been 
determined,  the  most  credible  subsequent  asset  configuration  trajectory  for  the  succeeding 
time  periods  may  be  computed  to  provide  a  provisional  road-map  for  the  anticipated  way 
ahead. 

Since  the  analysis  described  in  this  report  has  been  based  on  strictly  limited  informa¬ 
tion,  the  results  that  are  obtained  are  of  a  highly  tentative  nature.  Furthermore,  in  order 
to  employ  the  method  to  its  greatest  advantage,  the  ASBM  projects  would  need  to  be 
considered  in  conjunction  with  the  other  major  projects,  such  as  Air  6000  and  Sea  4000, 
with  which  they  are  intimately  intertwined.  Nevertheless,  the  potential  utility  of  such  an 
approach  is  convincingly  demonstrated. 
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1  Introduction 

The  problem  of  developing  and  sustaining  a  highly  capable  defence  force  to  meet  the  di¬ 
verse  needs  of  a  medium-sized  democratic  nation  such  as  Australia  is  an  extremely  onerous 
and  complex  one,  particularly  in  view  of  the  plethora  of  challenges  and  ambiguities  posed 
by  the  political  circumstances  of  the  twenty-first  century,  the  rapid  rate  of  technological 
advancement  of  military  systems,  and  the  relatively  modest  level  of  funding  available.  One 
way  of  maximising  the  potential  performance  of  a  defence  force  is  to  ensure  that  its  as¬ 
sets  are  capable  of  cooperating  cohesively  together  as  a  highly  effective  system-of-systems. 
This  goal  may  be  fostered  through  adopting  suitable  Force  Level  Systems  Engineering 
(FLSE)  principles. 

A  particularly  salient  feature  of  systems-of-systems  is  the  presence  of  multiple  stake¬ 
holders  and  decision-makers.  Sophisticated  and  robust  force-level  systems  design  methods 
that  can  effectively  match  the  needs  of  disparate  stakeholders  to  the  potentially  available 
options  are  required  in  order  to  alleviate  the  task  confronting  decision-makers.  However, 
certain  results  from  social  choice  theory  and  game  theory  have  implications  for  the  ways 
in  which  stakeholders’  preferences  may  validly  be  aggregated.  In  addition,  the  networked 
nature  of  systems-of-systems  renders  evaluation  methods  that  rely  on  a  strictly  hierar¬ 
chical  decomposition  of  the  decision  problem  unsuitable.  These  two  aspects  suggest  that 
Bayesian  Belief  Network  (BBN)  methods  may  be  more  suitable  for  treating  systems-of- 
systems  questions  than  are  more  familiar  evaluation  methods 

The  aim  of  this  report  is  to  illustrate  how  BBN  techniques  might  be  applied  to  assist  in 
making  FLSE  design  decisions.  The  particular  case  of  Aerospace  Surveillance  and  Battle 
Management  (ASBM)  is  considered.  The  techniques  in  question  have  been  introduced  in 
a  preceding  report  and  several  published  papers,  papers  [1,  2,  3].  Further  background  on 
BBN  techniques,  in  general,  may  be  obtained  from  the  books  by  Pearl  and  Neapolitan  [4, 
5]. 


1.1  Process 

The  method  to  be  described  is  intended  to  be  compatible  with  a  wide  variety  of  project 
management  and  systems  engineering  process  models,  and  consequently  there  is  no  par¬ 
ticular  preferred  process.  On  the  other  hand,  there  are  several  important  features  that  a 
process  suitable  for  treating  systems-of-systems  questions  ought  to  possess.  One  of  these  is 
that  there  should  be  far  more  assiduous  attention  to  the  human  dimensions  of  the  situation 
than  would  typically  be  the  case  in  a  more  conventional  systems  engineering  approach. 
Of  course,  it  is  necessary,  at  the  same  time,  to  ensure  that  the  objective  realities  of  the 
situation  are  properly  taken  into  account. 

In  dealing  with  systems-of-systems  questions,  the  presence  of  multiple  stakeholders 
and  decision-makers  is  a  fundamental  issue  that  needs  to  be  appropriately  accommodated. 
This  contrasts  with  conventional  systems  engineering  processes  where  it  is  generally  as¬ 
sumed  that  there  is  one  single,  predominant  decision-maker  whose  wishes  must  always  be 
paramount.  One  aspect  of  dealing  with  this  issue  is  to  ensure  that  fairness  and  trans¬ 
parency  are  provided  for  all  stakeholders.  This  assists  in  promoting  the  choice  of  the  best 
decisions  for  the  system-of-systems,  as  a  whole,  rather  than  allowing  the  available  resources 
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to  be  covertly  monopolised  by  the  most  influential  stakeholders,  to  the  detriment  of  total 
capability. 

A  further  important  feature  is  that  extensive  interaction  between  the  stakeholders 
should  be  catered  for  to  enable  tacit  and  implicit  knowledge  about  their  joint,  combined 
needs  to  be  incrementally  discovered  and  elicited.  Another  is  that  continuous  learning 
from  the  outcomes  of  past  decisions  should  be  supported  in  order  to  allow  the  quality  of 
decision-making  to  be  progressively  improved.  Other  features  which  should  be  considered 
are  facilitation  of  the  collaboration  of  geographically  dispersed  stakeholders,  and  effec¬ 
tive  management  of  large  quantities  of  complex  information,  in  order  to  allow  this  to  be 
efficiently  shared  and  utilised. 

In  general  terms,  the  method  involves  eliciting  the  needs  of  the  stakeholders  and  then 
generating  several  of  the  possible  solutions  that  would  seem  to  best  conform  to  those  needs 
for  more  intensive  consideration  by  stakeholders  and  decision-makers,  in  an  iterative  fash¬ 
ion.  Iteration  is  important  as  the  stakeholders’  understandings  of  their  individual  and 
shared  needs  may  mature  through  deeper  reflection  on  the  problem  and  the  proposed  so¬ 
lution  options.  Another  reason  for  the  importance  of  iteration  is  to  accommodate  changes 
in  the  stakeholders’  needs  that  may  arise  due  to  changing  circumstances. 

Naturally,  the  process  that  has  been  followed  in  producing  this  report,  which  merely 
seeks  to  demonstrate  the  capabilities  of  the  more  technical  portion  of  the  proposed  method 
in  a  realistic,  but  limited,  example,  is  substantially  different  from  any  which  might  be 
followed  in  an  actual  application. 


1.2  Information  Requirements 

The  information  that  would  normally  be  required  in  order  to  properly  apply  the  method 
to  be  described  here  is  listed  below. 

1.  A  list  of  the  stakeholders  affected  by  the  capability  under  investigation. 

2.  The  decision  epochs  to  be  considered. 

3.  The  needs  of  each  of  the  stakeholders  that  the  capability  will  be  expected  to  address, 
according  to  each  epoch  and  to  each  stakeholder. 

4.  Details  of  the  alternative  assets  that  will,  or  may  be,  available  for  procurement, 
according  to  each  epoch. 

5.  Details  about  supporting  data  links  that  might  be  required,  according  to  each  epoch. 

6.  Information  about  the  stakeholders’  preferences. 

For  items  4  and  5  some  measure  of  plausibility  is  required,  ideally  in  the  form  of  a 
credibility  figure.  Item  6  needs  to  be  captured  in  the  form  of  a  BBN  fragment.  It  may  be 
possible  to  glean  a  good  deal  of  the  necessary  information  from  high-level  systems  engi¬ 
neering  documents,  such  as  the  Operational  Concept  Documents  (OCDs)  for  the  systems 
concerned.  However,  this  will  generally  need  to  be  supplemented  with  direct  elicitation  of 
additional  information  from  the  stakeholders  and  relevant  experts. 
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Table  1:  Information  Sources 


No. 

Source 

Ref. 

1. 

Capability  Systems/Battlespace  Management  Project  List 

[6] 

2. 

Public  Release  Defence  Capability  Plan 

[7] 

3. 

DMO  Project  Web  Site 

[8] 

Table  2:  Relevant  Projects 


No. 

Area 

Project 

Title 

Abbrv. 

1. 

AIR 

5077 

Project  WEDGETAIL 

AEW&C 

2. 

AIR 

5333 

2  CRU  and  3  CRU  Replacement  (VIGILARE) 

Vigilare 

3. 

AIR 

5405 

Mobile  Sector  Operations  Centre 

MSOC 

4. 

JP 

117 

ADF  Ground-based  Air  Defence  Weapon  System 

GBAD 

5. 

JP 

2025 

Jindalee  Operational  Radar  Network 

JORN 

6. 

JP 

2044 

Space-based  Surveillance  Capability 

SBS 

7. 

JP 

2062 

Global  Hawk 

GH 

1.3  Information  Sources 

Unfortunately,  it  has  been  necessary  to  limit  the  information  on  which  this  report  is 
based  to  a  list  of  projects  relevant  to  ASBM,  the  public  Defence  Capability  Plan  (DCP), 
and  the  projects  web  site  maintained  by  the  Defence  Materiel  Organisation  (DMO).  Thus, 
a  considerable  amount  of  guesswork  has  been  required  in  order  to  be  able  to  generate  some 
results,  and,  consequently  any  conclusions  that  are  reached  must  be  regarded  as  being  of 
an  extremely  tentative  nature.  The  information  sources  that  were  employed  are  listed  in 
Table  1.  The  OCDs  for  several  of  the  systems  concerned  were  obtained  as  this  report  was 
being  finalised,  and  thus  at  a  stage  when  it  was  too  late  for  them  to  be  used  as  a  basis 
for  the  work.  However,  a  brief  perusal  of  them  would  suggest  that  the  guesses  made  in 
creating  the  example  used  in  this  report  were  not  too  wildly  divorced  from  reality. 

Source  1  lists  a  number  of  projects,  some  of  which  have  in-service  dates  around  the 
present  date,  and  some  of  which,  such  as  the  replacement  of  radar  transmitters  and  radio 
systems,  seem  to  relate  to  a  considerably  greater  level  of  detail  than  the  majority  of  the 
others.  Such  relatively  simple  upgrades  have  been  assumed  to  already  be  included  in  the 
Period  0  base-line  asset  configuration.  Some  of  the  other  projects  listed  are  very  vaguely 
described,  or  are  of  a  research  nature,  and  have  consequently  been  omitted  from  further 
consideration.  This  process  of  elimination  has  reduced  the  set  of  relevant  projects  for  the 
purposes  of  this  report  to  those  shown  in  Table  2. 


1.4  Problem  Formulation 

The  question  examined  in  this  report  is  the  most  credible  evolution  of  the  ASBM 
system-of-systems  under  tight  cost  constraints.  Certain  earliest  decision  dates  for  these 
projects  have  been  postulated  in  the  DCP  [7].  Here  it  is  assumed  that  it  is  possible  for 
the  decision  dates  to  be  postponed  beyond  those  stipulated  in  the  DCP,  but  that  it  is  not 
possible  to  bring  them  forward. 


3 


DSTO-TR-1475 


The  aim  is  to  characterise  the  ensemble  of  asset  configuration  trajectories  over  several 
periods  that  most  credibly  meet  the  stakeholders’  needs  under  these  circumstances,  and 
hence  to  identify  the  most  propitious  option  to  select  at  the  next  impending  decision 
epoch  by  means  of  credibilistic  inference.  A  decision  made  in  this  way  will  be  termed  a 
best  prospects  decision. 


2  Scenario 

2.1  Stakeholder  Preferences 

It  has  been  assumed  that  there  are  just  two  stakeholders  involved.  Stakeholder  A  may 
be  taken  to  represent  a  hypothetical  commander  who  may  have  to  conduct  a  campaign 
using  the  assets  that  are  provided.  Stakeholder  B  may  be  taken  to  represent  the  head  of 
finance  who  aspires  to  restrict  defence  expenditure  to  lie  within  affordable  limits. 

2.1.1  Stakeholder  A 

In  order  to  concord  with  the  list  of  projects  given  in  Table  2,  it  has  been  assumed  that 
the  needs  of  a  commander  that  should  be  provided  for  by  an  ASBM  system-of-systems  are 
the  following. 

1.  Air  space  surveillance. 

(a)  Long  range  surveillance. 

(b)  Medium  range  surveillance  —  lesser  surveillance  range  and  less  comprehensive 
coverage,  possibly  being  cued  by  long  range  surveillance,  but  increased  proba¬ 
bility  of  detection,  discrimination  and  accuracy. 

(c)  Local  surveillance  limited  to  line-of-sight  range  from  fixed  ground  locations, 
but  detailed,  comprehensive  and  precise. 

2.  Point  defence. 

3.  Airborne  communications  relay. 

2.1.2  Value  Network 

The  various  ways  in  which  the  individual  assets  contribute  towards  the  needs  of  the 
hypothetical  commander  are  represented  by  the  value  network  shown  in  Figure  1.  The 
network  approximates  to  the  form  of  a  tree,  with  each  asset  contributing  to  a  particu¬ 
lar  function.  However,  it  has  been  assumed  that  the  AEW&C  and  Global  Hawk  assets 
contribute  to  both  medium  range  air  surveillance  and  airborne  communications  relay  func¬ 
tions.  Thus,  the  network  is,  in  fact,  of  more  general  form.  Tables  A1-A13  in  Appendix  A 
give  the  numerical  credibility  values  that  are  used  to  reflect  the  relative  merits  of  the  assets 
under  consideration  in  performing  each  of  the  desired  functions. 
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2.1. 2.1  Long  Range  Air  Surveillance.  In  Table  Al,  it  has  been  assumed  that  SBS 
can  perform  all  the  functions  of  JORN  in  its  original  state,  and,  furthermore,  that  it  is 
about  80%  reliable.  On  the  other  hand  the  original  JORN  is  assumed  to  be  60%  reliable. 
The  upgraded  JORN  is  assumed  to  be  70%  reliable,  and  also  to  add  additional  capability 
beyond  that  provided  by  SBS  alone.  The  two  in  combination  are  assumed  to  be  90% 
reliable. 


2. 1.2. 2  Medium  Range  Air  Surveillance.  In  Table  A3,  it  has  been  assumed  that 
AEW&C  is  80%  reliable  in  performing  this  function  on  its  own,  while  Global  Hawk  on  its 
own  is  90%  reliable.  It  has  also  been  assumed  that  there  is  some  degree  of  complementarity 
between  the  two  so  that  the  two  combined  provide  95%  reliability. 


2. 1.2.3  Mobile  Operations.  In  Table  A5,  it  has  been  assumed  that  the  legacy  mobile 
sector  operations  centre  (114MCRU)  is  60%  reliable,  whereas  the  proposed  new  MSOC  is 
80%  reliable.  It  assumed  that  there  is  no  additional  advantage  in  having  both.  A  more 
realistic  model  might  assign  decreasing  reliability  with  time  for  legacy  assets,  but,  for  this 
simple  example,  this  has  not  been  done. 


2. 1.2.4  Local  Air  Surveillance.  In  Table  A6,  it  has  been  assumed  that  Vigilare 
alone  is  reliable  50%  of  the  time,  and  that  enhanced  Vigilare  is  reliable  60%  of  the  time. 
Combining  mobile  operations  with  Vigilare  is  assumed  to  give  reliabilities  of  70%  and  90% 
for  this  function. 


2. 1.2. 5  Weapons.  In  Table  A8,  it  has  been  assumed  that  the  legacy  air  defence 
weapons  are  60%  reliable,  while  the  GBAD  weapons  are  90%  reliable. 

2. 1.2. 6  Air  Surveillance.  In  Table  A9,  medium  range  surveillance  on  its  own  is  as¬ 
sumed  to  be  30%  reliable,  as  is  long  range  surveillance  on  its  own.  The  two  combined  are 
assumed  to  be  95%  reliable. 


2. 1.2.7  Airborne  Communications  Relay.  In  Table  A10,  AEW&C  on  its  own  is 
assumed  to  be  60%  reliable,  while  GH  on  its  own  is  assumed  to  be  70%  reliable.  The  two 
combined  are  assumed  to  be  90%  reliable.  Augmenting  the  number  of  AEW&C  improves 
the  reliability  accordingly. 


2. 1.2. 8  Point  Defence.  In  Table  A12,  local  air  surveillance  and  effective  weapons 
provide  90%  reliability.  Other  combinations  are  totally  ineffective. 

2. 1.2.9  Aerospace  Surveillance  and  Battle  Management.  In  Table  A13  the  asset 
configuration  is  assumed  to  be  100%  reliable  if  there  is  adequate  air  surveillance,  airborne 
communications  relay  and  point  defence.  It  is  assumed  to  be  30%  reliable  if  the  airborne 
communication  relay  function  is  inadequate.  Otherwise,  it  is  assumed  to  be  inadequate. 
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Table  3:  Cost  Categories  of  Project  Phases 


Cat. 

Projects 

1 

JORN(2),  MSOC,  Vigilare(2),  SBS(2) 

2 

SBS,  GH,  JORN(3) 

3 

AEW&C(2) 

4 

GBAD 

2.1.3  Stakeholder  B 

Stakeholder  B  is  assumed  to  be  increasingly  averse  to  total  procurement  costs  related 
to  ASBM  that  are  greater  than  $500,000  during  any  given  period.  The  project  phases 
have  been  grouped  into  cost  categories  by  rounding  the  figure  cited  in  the  DCP  up  to 
the  nearest  $100,000  [7].  Each  category  then  corresponds  to  some  multiple  of  $100,000. 
Where  a  figure  was  not  available  from  the  DCP,  some  guesswork  was  employed  to  deduce 
a  suitable  value.  The  resulting  cost  categories  for  each  project  phase  are  given  in  Table  3. 
In  addition  to  the  costs  of  upgrading  existing  assets  and  of  introducing  new  assets,  a  cost 
saving  of  $100,000  is  assumed  for  each  asset  that  is  retired. 

A  rough  estimate  of  the  total  cost  Ci  corresponding  to  decision  epoch  U  is  obtained 
by  summing  the  category  numbers  for  each  insertion  or  upgrade  choice,  and  subtracting 
one  for  each  retirement  choice.  In  the  case  of  GBAD,  the  procurement  cost  is  assumed  to 
be  spread  across  two  periods,  thus  two  units  of  cost  are  attributed  to  each,  rather  than 
attributing  all  four  units  to  the  period  immediately  following  the  decision  epoch. 

The  preferences  that  the  stakeholder  is  assumed  to  have  for  each  of  the  resultant  cost 
figures  are  given  in  Table  A14,  where  the  symbolic  labels  Co,  ...  ,cio  correspond  to  values 
of  $0-$ 1,000, 000.  It  can  be  seen  that  the  stakeholder  is  assumed  to  be  strongly  averse  to 
cost  values  of  greater  than  $500,000,  but  equanimous  towards  figures  below  this.  More 
realistically,  the  preferences  should  be  tapered  in  this  region,  however  this  was  not  done. 


2.2  Assets 

In  general,  there  is  a  fairly  direct  correspondence  between  projects  and  the  assets  they 
are  destined  to  procure.  However,  for  legacy  assets,  there  is  no  corresponding  project,  and 
hence  these  need  to  be  considered  separately.  The  legacy  assets  that  are  assumed  to  form 
part  of  the  ASBM  asset  configuration  are  114MCRU,  which  is  treated  as  a  legacy  mobile 
sector  operations  centre  (L.  MSOC),  and  legacy  air  defence  assets  (L.  Air  Defence). 


2.3  Initial  Conditions 

The  set  of  initial  conditions  specifies  the  circumstances  prevailing  at  the  outset,  in  par¬ 
ticular  the  base-line  configuration  of  the  assets.  For  the  ASBM  case,  the  initial  conditions 
are  assumed  to  be  those  shown  in  Table  4. 
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Table  /:  Baseline  Asset  Configuration 


Asset 

Condition^ ) 

AEW&C 

Absent 

Vigilare 

Present 

L.  MSOC 

Present 

L.  Air  Defence 

Present 

GBAD 

Absent 

JORN 

Original 

SBS 

Absent 

GH 

Absent 

Table  5:  Number  of  Decisions  in  each  Financial  Year 


FY 

03/04 

04/05 

05/06 

06/07 

07/08 

08/09 

09/10 

10/11 

Decisions 

1 

4 

0 

0 

2 

0 

0 

3 

2.4  Decision  Epochs 

In  order  to  reduce  the  complexity  of  the  problem  and  to  allow  the  “big  picture”  to 
emerge,  consideration  has  been  restricted  to  a  limited  number  of  decision  epochs.  Exam¬ 
ination  of  the  current  DCP  suggests  that  there  are  three  key  decision  epochs  for  the  set 
of  projects  in  question  [7].  Since  there  is  only  one  decision  for  FY  03/04,  this  can  most 
conveniently  be  combined  with  those  for  FY  04/05.  The  decision  epochs  that  result,  and 
the  projects  each  concerns,  are  listed  in  Table  6. 

The  choices  that  it  is  possible  to  make  at  each  of  these  decision  epochs  are  listed  in 
Table  7.  The  legacy  assets  are  assumed  to  have  reached  the  end  of  their  useful  life  by  epoch 
t3,  and  thus  are  unconditionally  retired  at  this  point.  To  concord  with  the  information  in 
the  DCP,  it  is  assumed  that,  although  the  earliest  that  the  decision  to  acquire  the  GBAD 
system  may  made  is  at  epoch  t\,  the  earliest  that  the  system  will  be  available  is  period 
•  No  matter  what  decision  is  made,  it  will  not  be  available  for  period  i/,  hence  there  is 
no  decision  to  be  made  concerning  it  at  epoch  1 2 . 


3  Results 

The  constraints  and  stakeholder  preferences  listed  above  were  manually  compiled  into 
the  form  of  a  BBN.  The  options  for  the  decision  epochs  are  assumed  to  be  constrained 


Table  6:  Selected  Decision  Epochs 


Epoch 

FY 

Decisions 

to 

02/03 

— 

t\ 

03/04,  04/05 

MSOC,  GBAD,  JORN(2),  SBS,  GH 

h 

07/08 

AEW&C(2),  Vigilare(2) 

h 

10/11 

SBS (2),  GH(2),  JORN(3) 
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Table  7:  Possible  Actions  by  Asset  and  Epoch 


Asset  /  Epoch 

ti 

h 

h 

AEW&C 

— 

f  Augment  by  2  1 
"j  Augment  by  3  r 
Withdraw  J 

f  Augment  by  2  I 
)  Augment  by  3  f 
l  Withdraw  J 

Vigilare 

Withdraw 

f  Withdraw  \ 

\  Enhance  J 

( Withdraw  \ 

\  Enhance  j 

L.  MSOC 

Withdraw 

Withdraw 

— 

MSOC 

Insert 

/  Insert  \ 

\  Withdraw  j 

f  Insert  \ 

\  Withdraw  j 

L.  Air  Defence 

Withdraw 

Withdraw 

— 

GBAD 

— 

— 

Withdraw 

JORN 

f  Upgrade  1 1 
|  Withdraw  J 

f  Upgrade  1 1 

X  Withdraw  J 

f  Upgrade  1  1 
)  Upgrade  2  f 
l  Withdraw  ) 

SBS 

Insert 

/  Insert  \ 

\  Withdraw  / 

(  Insert  | 

\  Upgrade  > 
l  Withdraw  J 

GH 

Insert 

f  Insert  1 
\  Withdraw  / 

f  Insert  \ 

\  Withdraw  j 
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only  by  the  asset  configuration  for  the  immediately  preceding  period.  In  this  example, 
no  additional  configurational  constraints  were  needed  because  each  of  the  assets  was  as¬ 
sumed  to  be  capable  of  operating  in  isolation  from  the  rest.  The  preferences  of  the  two 
stakeholders  were  given  equal  weight. 

The  value  network  of  Section  2.1.2  was  duplicated  three  times,  once  for  each  of  the 
periods  under  consideration.  Slight  modifications  were  made  each  time  to  reflect  the 
different  sets  of  asset  conditions  that  are  possible  for  each  period.  Value  network  fragments 
representing  Stakeholder  B’s  preferences  for  the  costs  associated  with  each  decision  epoch 
were  also  included.  The  resulting  network  appears  in  Figure  2.  The  “Netica”  BBN  software 
application  was  used  to  produce  the  results  presented  below  [9] . 


3.1  Marginal  Credibilities 

Figure  2  shows  the  marginal  credibilities  for  each  node  condition  in  the  network,  con¬ 
tingent  on  the  event  that  both  stakeholders  are  satisfied  for  each  decision  epoch  and  period. 
Note  that  the  credibility  figures  have  been  given  in  the  form  of  percentages.  The  results 
that  were  obtained  are  also  listed  in  Tables  B1-B9  in  Appendix  B  and  summarised  in 
colour-coded  form  in  Table  8.  A  key  for  the  colour  highlighting  used  in  Table  8  is  given 
in  Table  9. 

The  marginal  credibilities  provide  a  concise  characterisation  of  the  entire  ensemble  of 
asset  configuration  trajectories.  Appendix  C  provides  an  example  of  the  kind  of  sensitivity 
analysis  that  can  be  performed  for  individual  nodes. 


3.2  Most  Credible  Trajectory 

Unlike  computing  the  marginal  credibilities,  computing  the  most  credible  evolution  of 
the  force  structure  results  in  one  best  or  several  distinct,  equally  good,  asset  configuration 
trajectories.  In  this  example,  it  happens  that  a  unique  answer  is  obtained.  Figure  3  shows 
the  resulting  most  credible  asset  configuration  trajectory  into  the  future.  In  this  mode 
of  inference,  the  figures  for  the  asset  condition  credibilities  shown  on  the  diagram  do  not 
sum  to  one.  An  asset’s  condition  in  the  configuration  for  a  particular  period  is  the  one  for 
which  the  assigned  credibility  is  100%. 

The  result  is  also  presented  in  Table  10.  If  this  were  to  be  the  asset  configuration  tra¬ 
jectory  to  which  it  was  aspired,  then  the  decision  that  should  be  made  at  the  forthcoming 
decision  epoch  would  be  the  t\  option  for  this  particular  evolution.  However,  as  will  be 
explained  in  the  next  section,  there  are  some  arguments  in  favour  of  making  a  more  subtle 
selection. 

A  noteworthy  feature  of  this  solution  is  that  it  recommends  that  the  MSOC  be  with¬ 
drawn  for  period  2  and  re-inserted  for  period  3.  This  counter-intuitive  result  can  be 
attributed  to  the  large  amount  of  slack  in  the  expenditure  for  period  3.  This  means  that 
there  is  no  penalty  for  the  cost  of  re-inserting  the  asset.  As  remarked  earlier,  the  initial 
portion  of  the  cost  preference  curve  should,  more  realistically,  be  tapered  rather  than  flat. 
Also,  in  a  more  complete  model,  the  slack  expenditure  may  be  expected  to  be  taken  up 
with  other  projects. 
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Figure  2:  Belief  Network  showing  Marginal  Credibilities 
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Table  8:  Colour- coded  Marginal  Credibilities 


Asset  /  Period 

4 

4 

4 

4 

AEW&C 

[Absent 

[present 

Absent 

Resent 

|Absent 

Original 

Augmented  by  2 
Augmented  by  3 

Absent 
[briginal 
Augmented  by  2 
Augmented  by  3 

Vigilare 

ft 

Absent 

present 

jAbsent 

Original 

Enhanced 

{Absent 

Original 

Enhanced 

L.  MSOC 

Absent 

Absent 

Present 

Absent 

Present 

Absent 

Present 

MSOC 

Absent 

Present 

Absent 

Present 

Absent 

Present 

L.  Air  Defence 

Absent 

Present 

Absent 

present 

GBAD 

Present 

Present 

IjAbsent 

present 

Absent 

JORN 

Absent 
Original 
Upgraded  1 

Absent 

Upgraded  1 

Absent 

priginal 

Upgraded  1 
Upgraded  2 

SBS 

1 

Present 

Absent 

Present 

Absent 

Present 

Absent 

Original 

Upgraded 

GH 

] 

Present 

Absent 

Present 

Absent 

Present 

Absent 

Present 
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Table  9:  Colour  Key 


Range 

Colour 

o-o.i- 

0.1-0.2" 

«• 

i 

ft:  '  -  _ _ _ 

0.2-0.3- 

0.3-0.4" 

0.4-0.5- 

0. 5-0.6- 

0.6-0. 7“ 

0.7-0. 8“ 

0.8-0.9- 

t: 

0.0-1. 0- 

i- ,  . 

1.0 

MmWf- 

Table  10:  Most  Credible  Trajectory 


Asset 

4 

4 

4 

4 

AEW&C 

Absent 

Present 

Augmented  by  3 

Augmented  by  3 

Vigilare 

Present 

Present 

Enhanced 

Enhanced 

L.  MSOC 

Present 

Absent 

Absent 

Absent 

MSOC 

Absent 

Present 

Absent 

Present 

L.  Air  Defence 

Present 

Present 

Present 

Absent 

GBAD 

Absent 

Absent 

Present 

Present 

JORN 

Present 

Absent 

Absent 

Absent 

SBS 

Absent 

Present 

Present 

Upgraded 

GH 

Absent 

Present 

Present 

Present 
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Figure  3:  Belief  Network  showing  Most  Credible  Trajectory 
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Table  11:  Best  Prospects  Decision 


Asset 

Action(ti) 

AEW&C 

Vigilare 

L.  MSOC 
MSOC 

L.  Air  Defence 
GBAD 

JORN 

SBS 

GH 

None 

None 

Insert 

None 

Insert 

None 

None 

Insert 

3.3  Best  Prospects  Decision 

It  may  reasonably  be  assumed  that  it  is  only  ever  necessary  to  actually  commit  to  an 
option  for  the  first,  impending  decision  epoch.  The  best  prospects  decision  is  defined  here 
as  the  selection  of  the  most  credible  option  for  the  first  decision  epoch  t\  over  the  entire 
ensemble  of  asset  configuration  trajectories.  This  option  is  not,  in  general,  the  same  as 
the  ti  option  for  the  most  credible  trajectory. 

The  reason  for  recommending  the  best  prospects  option,  rather  than  the  t\  option  for 
the  most  credible  trajectory,  is  that  choosing  this  should  usually  result  in  there  being  more 
relatively  high  credibility  alternative  trajectories  on  into  the  future,  thus  providing  greater 
flexibility  as  stakeholder  needs  and  preferences  undergo  future  revision.  Consequently, 
the  best  prospects  decision  sacrifices  some  initial  credibility  about  the  alternative  that  is 
selected  in  the  interests  of  greater  future  flexibility.  Unfortunately,  the  software  that  was 
used  does  not  support  the  direct  calculation  of  this  option.  However,  with  some  additional 
manual  manipulation  it  is  possible  to  find  it  fairly  readily.  The  best  prospects  option  that 
was  obtained  in  this  way  is  given  in  Table  11. 

Given  a  best  prospects  decision  for  t\,  it  is  then  of  interest  to  determine  the  most 
credible  subsequent  asset  configuration  trajectory  in  order  to  provide  a  provisional  road¬ 
map  for  the  way  ahead.  The  BBN  in  which  this  calculation  is  performed  is  shown  in 
Figure  4,  and  the  results  are  summarised  in  Table  12.  The  asset  conditions  which  differ 
from  those  for  the  most  credible  trajectory  overall  are  shown  emphasised  thus.  This 
trajectory  is  less  credible  than  the  most  credible  trajectory  by  a  factor  of  7.  For  the  best 
prospects  decision  to  be  worthwhile,  this  figure  would  need  to  be  less  than  the  fluctuations 
in  credibility  anticipated  to  be  brought  about  by  future  changes  in  the  value  networks. 


4  Conclusion 

This  report  has  provided  an  example  of  the  application  of  a  Force  Level  Systems 
Engineering  decision  support  method  based  on  Bayesian  Belief  Network  techniques  to 
a  collection  of  projects  related  to  Aerospace  Surveillance  and  Battle  Management.  It 
illustrates  the  kind  of  decision  problem  that  can  be  solved  by  such  means,  and  the  types 
and  quantity  of  information  required  to  do  so. 
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Figure  4:  Belief  Network  showing  Most  Credible  Trajectory  for  Best  Prospects  Decision 
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Table  12:  Most  Credible  Trajectory  for  Best  Prospects  Decision 


Asset 

4 

tf 

4 

4 

AEW&C 

Absent 

Present 

Augmented  by  3 

Augmented  by  3 

Vigilare 

Present 

Present 

Enhanced 

Enhanced 

L.  MSOC 

Present 

Present 

Absent 

Absent 

MSOC 

Absent 

Present 

Absent 

Present 

L.  Air  Defence 

Present 

Present 

Present 

Absent 

GBAD 

Absent 

Absent 

Present 

Present 

JORN 

Present 

Original 

Absent 

Absent 

SBS 

Absent 

Absent 

Present 

Upgraded 

GH 

Absent 

Present 

Present 

Present 

However,  given  the  severely  limited  information  on  which  the  analysis  was  based,  the 
results  that  were  obtained  are  of  a  highly  tentative  nature.  The  more  timely  availability 
of  more  detailed  information  might  have  permitted  the  development  of  a  more  realistic 
model  and,  consequently,  might  have  yielded  results  in  which  a  greater  degree  of  confidence 
could  be  held.  The  more  comprehensive  information  that  has  since  become  available  would 
also  need  to  be  supplemented  with  some  direct  elicitation  of  the  pertinent  stakeholders’ 
preferences  in  order  to  properly  apply  the  method. 

Furthermore,  to  employ  the  method  to  its  greatest  advantage,  the  Aerospace  Surveil¬ 
lance  and  Battle  Management  projects  would  need  to  be  considered  in  conjunction  with 
the  other  major  projects,  such  as  Air  6000  and  Sea  4000,  with  which  they  are  intimately 
intertwined. 
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Appendix  A  Value  Tables 

This  appendix  contains  tables  listing  the  conditional  credibility  values  used  in  the  value 
networks. 


Table  Al:  Long  Range  Air  Surveillance  (LRS),  Periods  1&2 


JORN 

SBS 

Inadequate 

Adequate 

Absent 

Absent 

1 

0 

Absent 

Present 

0.2 

0.8 

Original 

Absent 

0.4 

0.6 

Original 

Present 

0.2 

0.8 

Upgraded  1 

Absent 

0.3 

0.7 

Upgraded  1 

Present 

0.1 

0.9 

Table  A2:  Long  Range  Air  Surveillance  (LRS),  Period  3 


JORN 

SBS 

Inadequate 

Adequate 

Absent 

Absent 

1 

0 

Absent 

Original 

0.4 

0.6 

Absent 

Upgraded 

0.3 

0.7 

Original 

Absent 

0.5 

0.5 

Original 

Original 

0.25 

0.75 

Original 

Upgraded 

0.2 

0.8 

Upgradedl 

Absent 

0.4 

0.6 

Upgradedl 

Original 

0.2 

0.8 

Upgradedl 

Upgraded 

0.1 

0.9 

Upgraded2 

Absent 

0.3 

0.7 

Upgraded2 

Original 

0.1 

0.9 

Upgraded2 

Upgraded 

0.05 

0.95 

Table  A3:  Medium  Range  Air  Surveillance  (MRS),  Period  1 


GH 

AEWC 

Inadequate 

Adequate 

Absent 

Absent 

1 

0 

Absent 

Present 

0.2 

0.8 

Present 

Absent 

0.1 

0.9 

Present 

Present 

0.05 

0.95 
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GH 

AEWC 

Inadequate 

Adequate 

Absent 

Absent 

1 

0 

Absent 

Original 

0.6 

0.4 

Absent 

Augmented  by  2 

0.4 

0.6 

Absent 

Augmented  by  3 

0.3 

0.7 

Present 

Absent 

0.7 

0.3 

Present 

Original 

0.2 

0.8 

Present 

Augmented  by  2 

0.1 

0.9 

Present 

Augmented  by  3 

0.05 

0.95 

2&3 


Table  A5:  Mobile  Operations  (MO) 


L_MSOC 

MSOC 

Inadequate 

Adequate 

Absent 

Absent 

1 

0 

Absent 

Present 

0.2 

0.8 

Present 

Absent 

0.4 

0.6 

Present 

Present 

0.2 

0.8 

Table  A  6:  Local  Air  Surveillance  (LAS),  Period  1 


VIGILARE 

MO 

Inadequate 

Adequate 

Absent 

Inadequate 

1 

0 

Absent 

Adequate 

0.6 

0.4 

Present 

Inadequate 

0.4 

0.6 

Present 

Adequate 

0.1 

0.9 

Table  A7:  Local  Air  Surveillance  (LAS),  Periods  2&3 


MO 

VIGILARE 

Inadequate 

Adequate 

Inadequate 

Absent 

1 

0 

Inadequate 

Original 

0.5 

0.5 

Inadequate 

Enhanced 

0.4 

0.6 

Adequate 

Absent 

1 

0 

Adequate 

Original 

0.3 

0.7 

Adequate 

Enhanced 

0.1 

0.9 

Table  A8:  Weapons  (W) 


GBAD 

L_Air_Defence 

Inadequate 

Adequate 

Absent 

Absent 

1 

0 

Absent 

Present 

0.4 

0.6 

Present 

Absent 

0.1 

0.9 

Present 

Present 

0.1 

0.9 
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Table  A9:  Air  Surveillance  (AS) 


LRS 

MRS 

Inadequate 

Adequate 

Inadequate 

Inadequate 

1 

0 

Inadequate 

Adequate 

0.7 

0.3 

Adequate 

Inadequate 

0.7 

0.3 

Adequate 

Adequate 

0.05 

0.95 

Table  A10:  Airborne  Communications  Relay  (ACR),  Period  1 


GH 

AEWC 

Inadequate 

Adequate 

Absent 

Absent 

1 

0 

Absent 

Present 

0.4 

0.6 

Present 

Absent 

0.3 

0.7 

Present 

Present 

0.1 

0.9 

Table  All:  Airborne  Communications  Relay  (ACR),  Periods  2&S 


GH 

AEWC 

Inadequate 

Adequate 

Absent 

Absent 

1 

0 

Absent 

Original 

0.4 

0.6 

Absent 

Augmented  by  2 

0.3 

0.7 

Absent 

Augmented  by  3 

0.2 

0.8 

Present 

Absent 

0.3 

0.7 

Present 

Original 

0.1 

0.9 

Present 

Augmented  by  2 

0.05 

0.95 

Present 

Augmented  by  3 

0 

1 

Table  A 12:  Point  Defence  (PD) 


LAS 

W 

Inadequate 

Adequate 

Inadequate 

Inadequate 

1 

0 

Inadequate 

Adequate 

1 

0 

Adequate 

Inadequate 

1 

0 

Adequate 

Adequate 

0.1 

0.9 

Table  A 13:  Aerospace  Surveillance  and  Battle  Management  (ASBM) 


AS 

ACR 

PD 

Inadequate 

Adequate 

Inadequate 

Inadequate 

Inadequate 

1 

0 

Inadequate 

Inadequate 

Adequate 

1 

0 

Inadequate 

Adequate 

Inadequate 

1 

0 

Inadequate 

Adequate 

Adequate 

1 

0 

Adequate 

Inadequate 

Inadequate 

1 

0 

Adequate 

Inadequate 

Adequate 

0.7 

0.3 

Adequate 

Adequate 

Inadequate 

1 

0 

Adequate 

Adequate 

Adequate 

0 

1 
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Appendix  B  Marginal  Credibilities 

This  appendix  contains  tables  listing  the  marginal  credibilities  of  the  conditions  of 
each  asset  by  period.  Since  the  figures  are  approximate,  excess  significant  figures  should 
be  disregarded. 


Table  Bl:  A5077  AEW&C  Marginal  Credibilities 


Condition 

4 

4 

4 

4 

Absent 

1 

0 

0.057579 

0.23158 

Original 

0 

1 

0.33899 

0.091322 

Augmented  by  2 

0 

0 

0.307 

0.33863 

Augmented  by  3 

0 

0 

0.29643 

0.33847 

Table  B2:  A5333  Vigilare  Marginal  Credibilities 


Condition 

4 

4 

4 

4 

Absent 

0 

0 

0 

0 

Original 

1 

1 

0.53362 

0.2464 

Enhanced 

0 

0 

0.46638 

0.7536 

Table  B3:  LI  Legacy  MSOC  Marginal  Credibilities 


Condition 

4 

4 

"TF 

z2 

4 

Absent 

0 

0.5099 

0.76494 

1 

Present 

1 

0.4901 

0.23506 

0 
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Table  B4-'  J- 

5405  MSOC  Marginal  Credibilities 

Condition 

E9 

4 

4 

4 

Absent 

1 

0.47434 

0.47317 

0.43294 

Present 

0.52566 

0.52683 

0.56706 

Table  B5:  L2  Legacy  Air  Defence  Marginal  Credibilities 


Condition 

/  + 
l0 

4 

4 

4 

Absent 

0 

0 

0.44767 

1 

Present 

1 

1 

0.55233 

0 

Table  B6:  J117  GBAD  Marginal  Credibilities 


Condition 

*0 

4 

4 

4 

Absent 

1 

1 

0 

0 

Present 

0 

0 

1 

1 

Table  B7:  J2025  JQRN  Marginal  Credibilities 
r^nnHit.irvn  I  I  7^"  TT~F  I  7T 


Condition 

4 

4 

4 

*3 

Absent 

0 

0.25301 

0.48596 

0.6111 

Original 

1 

0.40872 

0.16354 

0.04358 

Upgradedl 

0 

0.33827 

0.3505 

0.18154 

Upgraded2 

0 

0 

0 

0.16379 

Table  B8: 

J2044  SBS  Marginal  Credibilities 

Condition 

4 

4 

4 

Absent 

n 

0.47294 

0.39141 

0.26826 

Original 

0.52706 

0.60859 

0.23831 

Upgraded 

0.49344 

Table  B9:  J2062  GH  Marginal  Credibilities 


Condition 

ca 

4 

4 

4 

Absent 

1 

0.45979 

0.40981 

0.35398 

Present 

0 

0.54021 

0.59019 

0.64602 
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Appendix  C  Sensitivity  Analysis  for  Node 

D_J2025_1 

This  appendix  provides  an  example  of  the  kind  of  sensitivity  analysis  that  may  be 
performed.  The  target  variable  that  has  been  chosen  for  this  example  is  decision  taken 
on  JORN  at  epoch  1,  which  has  the  designation  D_J2025_1.  The  numerical  column  on 
the  left  shows  the  mutual  entropy  reduction,  in  bits,  that  can  be  achieved  by  a  finding 
for  the  second  variable  in  the  list.  The  column  on  the  right  shows  the  relative  decrease  in 
variance  that  may  be  achieved. 

The  results  show  that  the  variable  that  has  most  influence,  outside  the  trivial  case  of 
itself,  is  the  condition  of  the  asset  during  period  1.  This  must  self-evidently  be  the  case, 
since  the  condition  of  the  asset  is  a  deterministic  function  of  the  decision  variable.  The 
next  most  influential  variables  are  the  condition  and  decision  variables  for  the  same  asset 
for  subsequent  periods  and  epochs.  After  these,  as  might  be  expected,  the  cost  for  period 
1  becomes  most  significant.  Further  down  the  list  are  variables  connected  with  related 
projects,  such  as  space-based  surveillance  (J2044)  and  Global  Hawk  (J2062). 

Hence,  it  is  apparent  that  sensitivity  analysis  such  as  this  can  provide  some  useful 
insights  into  the  most  important  connections  between  the  various  projects.  These  connec¬ 
tions  might  be  useful  in  suggesting  groups  of  projects  that  should  lie  within  the  realm  of 
responsibility  of  a  single  chain  of  management,  for  example. 


Sensitivity  of  ,D_J2025_1’  due  to  a  finding  at  another  node: 


Node 

Mutual 

Variance  < 

— 

Info 

Beliefs 

J2025_l 

1.55821 

0.4321961 

D_J2025_1 

1.55821 

0.4321961 

J2025_2 

0.50144 

0.0945923 

D_J2025_2 

0.34608 

0.0638003 

J2025_3 

0.25947 

0.0304327 

D_J2025_3 

0.22037 

0.0268885 

C_1 

0 . 07364 

0.0111973 

D_J2044_1 

0.06979 

0.0073491 

J2044_l 

0.06979 

0.0073491 

LRS 

0.03130 

0.0048546 

J2044_2 

0.02182 

0.0024708 

D_J2044_2 

0.02032 

0.0023851 

LRS1 

0.01823 

0.0027606 

J2044.3 

0.01335 

0.0015787 

D_J2062_1 

0.01074 

0.0013785 

J2062_l 

0.01074 

0.0013785 

D_J2044_3 

0.01005 

0.0013044 

LRS  2 

0.00887 

0.0012690 

C_3 

0.00810 

0.0009089 

D_J2062_2 

0.00546 

0.0007227 
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C_2 

0.00477 

0.0005939 

D_L1_1 

0.00199 

0.0003539 

Ll_l 

0.00199 

0.0003539 

A5405_l 

0.00160 

0.0002951 

D_A5405_1 

0.00160 

0.0002951 

D_A5405_2 

0.00087 

0.0001648 

MRS2 

0.00075 

0.0000895 

MRS1 

0.00072 

0.0000799 

Ll_2 

0.00071 

0.0001130 

D_L1_2 

0.00069 

0.0001293 

MRS 

0.00066 

0.0000751 

ACR 

0.00056 

0.0000742 

D_A5077_2 

0.00052 

0.0000620 

A5077_2 

0.00052 

0.0000620 

A5077_3 

0.00052 

0.0000645 

MO 

0.00028 

0.0000311 

ACR2 

0.00016 

0.0000191 

D_A5077_3 

0.00010 

0.0000126 

J2062_3 

0.00009 

0.0000119 

L2_2 

0.00009 

0.0000113 

D_L2_2 

0.00009 

0.0000113 

MOl 

0.00006 

0.0000084 

A5333_3 

0.00005 

0.0000066 

ACR1 

0.00004 

0.0000050 

J2062.2 

0.00004 

0.0000049 

A5405_3 

0.00004 

0.0000048 

A5333_2 

0.00004 

0.0000045 

D_A5333_2 

0.00004 

0.0000045 

MO  2 

0.00003 

0.0000035 

D_J2062_3 

0 . 00002 

0.0000036 

A5405.2 

0.00001 

0.0000015 

D_A5405_3 

0.00001 

0.0000007 

D_A5333_3 

0.00000 

0.0000000 

PD2 

0.00000 

0.0000000 

ASBM1 

0.00000 

0.0000000 

AC_1 

0.00000 

0.0000000 

AS1 

0.00000 

0.0000000 

L2_i 

0.00000 

0.0000000 

ASBM2 

0.00000 

0.0000000 

D_A5333_1 

0.00000 

0.0000000 

AS2 

0.00000 

0.0000000 

A5333.1 

0.00000 

0.0000000 

D_J117_1 

0.00000 

0.0000000 

D_L2_1 

0.00000 

0.0000000 

LAS 

0.00000 

0.0000000 

PD1 

0.00000 

0.0000000 

W1 

0.00000 

0.0000000 
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PD 

0.00000 

0.0000000 

LAS  2 

0.00000 

0.0000000 

J117.2 

0.00000 

0.0000000 

AC_2 

0.00000 

0.0000000 

D_J117_3 

0.00000 

0.0000000 

W2 

0.00000 

0.0000000 

AC_3 

0.00000 

0.0000000 

J117_3 

0.00000 

0.0000000 

ASBM 

0.00000 

0.0000000 

AS 

0.00000 

0.0000000 

LAS1 

0.00000 

0.0000000 

W 

0.00000 

0.0000000 
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